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(54) Information processing and mixed reality presentation 



(57) A view transfomnation matrix that represents 
the position/attitude of an HMD Is generated based on 
a signal that represents the position/attitude of the IHMD 
(8602). On the other hand, landmarks and their loca- 
tions are detected based on a captured picture (S604) 
and a calibration matrix AMc is generated using the de- 
tected locations of the landmarks (S605). The position/ 



attitude of the HMD is calibrated using the view trans- 
formation matrix and calibration matrix AMc generated 
by the above processes (S606), a picture of a virtual ob- 
ject is generated based on external parameters that rep- 
resent the position/attitude of the calibrated HMD, and 
a mixed reality picture Is generated (S607). The gener- 
ated mixed reality picture Is displayed In the display sec- 
tion (S609). 
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Description 

FIELD OF THE INVENTION 

5 (0001 1 The present Invention relates to an Information processing apparatus that derives the calibration information 
needed to measure the position and/or attitude of a measuring object based on the output values of a position and/or 
attitude sensor, a mixed reality presentation apparatus that displays virtual space superimposed over a picture of 
captured real space on a display screen or displays virtual space superimposed over the real space transmitted optically 
through a display screen, a method thereof, and a storage medium. 

10 

BACKGROUND OF THE INVENTION 

[0002] Recently, studies have been conducted actively on mixed reality (hereinafter abbreviated to MR). MR aims 
at seamless connection between real space and virtual space. It is an Important technique In virtual reality (hereinafter 
15 abbreviated to VR). 

[0003] MR aims at coexistence of real space and a VR worid which can conventionally be experienced only In situ- 
ations isolated from real space. It attracts attention as a technique for enhancing VR. 

[0004] MR includes video see-through mode that Involves superimposing a picture of virtual space (for example, 
virtual objects drawn by computer graphics (hereinafter abbreviated to CG) or text Information) over a picture of cap- 
20 tured real space captured by an imaging apparatus such as a video camera and optical see-through mode that involves 
superimposing a picture of virtual space over a picture of the real space transmitted optically through a display screen. 
[0005] MR Is expected to find applications In medical aids which will show the inside of the patienfs body to the 
surgeon as if It were seen through, applications In woric aids which will show assembly procedures of a product super- 
Imposed over the actual objects in a factory, and other applications totally different in quality from those of conventional 

25 VR. 

[0006] What Is commonly required of these applications Is a technique for aligning the position/attitude of real space 
and virtual space and many attempts have been made so far 

[0007] The problem of position/attitude alignment in video see-through mode MR resolves itself down to a matter of 
finding the 3D position/attitude of a video camera in a global coordinate system set up In real space (hereinafter referred 
30 to simply as a global coordinate system). Similarly, the problem of position/attitude alignment In optical see-through 
mode MR comes down to a matter of finding the 3D position/attitude of the user's viewpoint. 
[0008] As a method of solving these problems, It is common practice to derive the 3D position/attitude of the video 
camera's or user's viewpoint In a global coordinate system by using a 3D position/attitude sensor such as a magnetic 
sensor or ultrasonic sensor. 

35 [0009] The output values of a sensor represent, In the case of a magnetic sensor, for example, the 3D position/ 
attitude of the sensor itself in the sensor coordinate system whose center is defined as the source of an AC magnetic 
field described later, and not the 3D position/attitude of the video camera's or user's viewpoint In a global coordinate 
system being measured. In other words, the output values of the sensor cannot be used directly as the 3D position/ 
attitude of the video camera's or user's viewpoint, but some coordinate transfomiation Is necessary. Hereinafter the 

40 data needed for coordinate transfomnation between the output values of the sensor and the 3D position/attitude of the 
video camera's or user's viewpoint will be referred to as calibration data (calibration Infonnatlon). Also, the process of 
setting or calculating calibration data beforehand will be refenred to as calibration. Only after accurate calibration is 
performed, the output of a 3D position/attitude sensor can be converted to the 3D position/attitude of the video camera's 
or user's viewpoint and accurate position/attitude alignment in MR can be pertomned. 

45 [0010] Conventionally, however, It Is difficult to: 

. acquire the position/attitude (the Information needed to convert sensor output to the 3D position/attitude of the 
sensor in a global coordinate system) of the sensor coordinate system in a global coordinate system, and 

. acquire the position/attitude (the infomiation needed to transf ornn the position/attitude of the sensor Into the posl- 
50 tlon/attltude of the measuring object) of the measuring object (camera's or user's viewpoint) as viewed from the 
sensor. 

[0011] The present invention has been made in view of the above problems. Its object is to acquire the Infomnation 
needed to transform a sensor coordinate system Into a global coordinate system and the Infomiation needed to trans- 
55 fornn the position/attitude of a sensor Into the position/attitude of a measuring object. 
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SUMMARY OF THE INVENTION 

[0012] To achieve the object of the present Invention, an Information processing apparatus according to the present 
invention, for example, has the following configuration. 
5 [0013J Specifically, it is an Information processing apparatus that derives the calibration infonmation needed to meas- 
ure the position and/or attitude of a measuring object based on the output values of a position and/or attitude sensor, 
comprising: 

Input means for entering Infomriatlon about a match between the position and/or attitude of the above described 
10 measuring object and a predetennined position and/or attitude; 

acquisition means for acquiring the output values from the above described position and/or attitude sensor ac- 
cording to the input by the above described input means; and 

operation means for deriving the above described calibration Information, based on the above described prede- 
termined position and/or attitude and the output values of the above described position and/or attitude sensor 
15 acquired by the above described acquisition means. 

[0014] To achieve the object of the present invention, a mixed reality presentation apparatus according to the present 
invention, for example, has the following configuration. 

[001 5] Specifically, it is a mixed reality presentation apparatus that displays virtual space superimposed over a picture 
20 of captured real space on a display screen or displays virtual space superimposed over the real space transmitted 
optically through a display screen based on output values of a position and/or attitude sensor, comprising: 

an information processing apparatus recited in any of claints 10 to 16; and 

switching means for switching between presentation mode that presents mixed reality and derivation mode that 
25 derives calibration infomnation; wherein 

the calibration Infonnation needed for presentation of the above described mixed reality is derived in the above 
described derivation mode and mixed reality is presented using the calibration infonnation In the above described 
presentation mode. 

30 [001 6] Other features and advantages of the present Invention will be apparent from the following description taken 
in conjunction with the accompanying drawings. In which like reference characters designate the same or similar parts 
throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

J5 

[0017] The accompanying drawings, whteh are incorporated In and constitute a part of the specification, illustrate 
embodiments of the invention and, together with the description, serve to explain the principles of the Invention. 

FIG. 1 1s a side view of the game device of the system according to a first embodiment of the present invention; 
40 FIG. 2 is a drawing showing a picture as viewed by left player 2000 through his/her own HMD 21 OL; 
FIG. 3 is a drawing showing the configuration of HMD 210; 
FIG. 4 is a drawing illustrating an outline of calibration setup; 

FIG. 5 Is a drawin g showing detailed configuration of the calibration section 5055L according to the first embodiment 
of the present invention; 

45 FIG. 6 Is a process flowchart of the calibration apparatus according to the first embodiment of the present Invention; 

FIG. 7A is a drawing showing the Image displayed on a display section 210L in Step SI 040; 
FIG. 78 is a drawing showing a picture optically transmitted through the display section 21 OL and viewed by a 
player, 

FIG. 7C is a drawing showing how the positions of a virtual mariner picture and a real marker picture are superim- 
50 posed exactly by parallel translation or rotation of HMD 21 OL (i.e., viewpoint 1901) in Step S1070; 

FIG. 8 is a drawing showing the configuration of a 3D image generation/presentation system that utilizes the cal- 
ibration data obtained through calibration on the game device of FIG. 1 ; 

FIG. 9 is a flowchart showing the control procedures for measurement of mallet position; 

FIG. 1 0 is a flowchart showing the control procedures for measurement of mallet position; 
55 FIG. 11 is a flowchart showing the control procedures for measurement of mallet position; 

FIG. 12 is a drawing Illustrating the split halves of the Image obtained by a CCD camera 230 located In a fixed 

position; 

FIG. 13 Is a drawing showing a search area; 
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FIG. 1 4 is a drawing showing the game field for air hockey games according to the first embodiment of the present 

Invention; 

FIG. 15 is a flowchart illustrating the processing procedures carried out in a game status control section 5030; 
FIG. 1 6 is a drawing illustrating the definitions of width and height; 
5 FIG. 1 7 is a flowchart of processes carried out in position/attitude transformation sections 5060 (5060L, 5060R) 

and image generation sections 5050 (5050L, 5050R); 

FIG. 18 Is a conceptual diagram Illustrating how a magnetic sensor measures Its position/attitude; 
FIG. 19 is a drawing illustrating how the position/attitude transfomiation section 5060L determines the position/ 
attitude Mvw of the viewpoint 1901 of the player 2000 in a global coordinate system 8000; and 
10 FIG. 20 is a drawing showing the equipment configuration used for the calibration of the game device according 

to the first embodiment of the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

15 [0018] Preferred embodiments of the present invention will now be described In detail in accordance with the ac- 
companying drawings. 

[First Embodiment] 

20 [0019] Now detailed description will be given about cases In which the Infomnatlon processing apparatus and method 
thereof according to the present Invention are applied to the calibration apparatus and method thereof for the calibration 
of a game device, and furthemnore, for example, to an air hockey game device that utilizes MR technology. 
[0020] The air hockey game is a game which is played between two opposing players who try to score goals by 
shooting a puck floated by compressed air supplied from below, Into the opponent's cage. The one who scores more 

25 goals wins. In the air hockey game that uses the MR technology of this embodiment, the puck is presented as a virtual 
3D image superimposed over a table in the real environment to the players, who compete to hit It with real mallets. 

<Configuration of the game device> 

30 [0021] FIG. 1 1s a side view of the game device of the system according to this embodiment. In the air hockey game 
that uses the MR technology, two players 2000 and 3000 face each other across a table 1000 with a mallet (260L or 
260R) in their hand. The two players 2000 and 3000 wear head-mounted displays 21 OL and 21 OR (hereinafter abbre- 
viated to HMD), respectively, on their head. The mallets (260L and 260 R) of this embodiment have an infrared emitter 
at their tip. The infrared rays emitted from the Infrared emitter are used to detect the positions of the mallets. Details 

35 will be described later. Incidentally, If the mallets have distinctive shapes or colors, their positions may be detected 
through pattern recognition using these distinctive features. 

[0022] The HMDs 210L and 210R of this embodiment are a see-through type as shown in FIG. 3, They will be 
described In detail later. The two players 2000 and 3000 can see the surface of the table even if they wear the HMD 
21 OL or 21 OR. Also, the HMDs 21 OL and 21 OR receive 3D virtual Images from an image processing system described 
40 later. Therefore, the players 2000 and 3000 view a 3D virtual pteture displayed on the display screen of the HMD 21 OL 
or 21 OR and superimposed over the real space transmitted through the optical system (not shown in FIG. 1) of the 
HMD 210L or210R 

[0023] FIG. 2 Is a drawing showing a picture as viewed by the left player 2000 through his/her own HMD 21 OL. The 
two players compete to shoot a puck 1500 represented by a virtual picture. The player 2000 uses the real mallet 260L 
45 held in his/her hand to hit the puck. The goal 1 200 R Is visible in front of the opposition player 3000. The image processing 
system (not shown in FIG. 2) generates a three-dimensional CG and displays it on the HMD 21 OL so that the goal 
1200R will be visible near the opponent. 

[0024] The opposition player 3000 will also see a virtual goal (not shown) near the player 2000 through the HMD 
21 OR. The puck 1500 Is also generated by the image processing system (not shown) and displayed on each HMD 

50 

<HMD with a magnetic sensor> 

[0025] FIG. 3 shows the configuration of the HMD 210. This HMD 210 Is the see-through type. It Is. for example the 
HMD body disclosed by Japanese published unexamined application H7-333551 with a magnetic sensor 220 mounted 
55 via a support rod 221 . Reference numeral 21 1 denotes an LCD panel. The light (the pteture presented to the observer) 
from the LCD panel 21 1 enters an optical member 21 2, reflects off a totally reflective surface 21 4, reflects off the totally 
reflective surface of a concave minror 21 3, and reaches the observer's eyes passing through the totally reflective surface 
214. 
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[0026] The magnetic sensor 220 measures the position/attitude of the observer's viewpoint. Since the magnetic 
sensor Is vulnerable to magnetic noise, it Is separated from the LCD panel 211 , which Is a noise source, by the support 
rod 221. 

[0027] Incidentalty. the configuration in which a magnetic sensor 220 is mounted on an HMD 21 0 as shown in FIG. 
5 3 is not limited to optical see-through type HIVIDs. It may also be applied to video see-through type HIVIDs of for the 
purpose of detecting the position/attitude of the video camera's viewpoint. 

[0028] In FIG. 1 , the HMDs 21 OL and 21 OR are secured to the head of the players 2000 and 3000, respectively, with 
a band (not shown). The players have the magnetic sensor 220 (220L or 220R) shown in FIGS. 1 and 3 as well as a 
CCD camera 240 (240L or 240R), imaging means, fastened to their head. The field of view of the CCD camera 240 Is 
10 set to the area in front of the player. In the case of air hockey games, since the players look at the top surface of the 
table 1000, the cameras 240 also capture the surface of the table 1000. An AC magnette field generator 250 is fixed 
at any such place that the magnetic sensors 220 fall within Its range. The magnetic sensors 220 sense the changes 
in the AC magnetic field generated by the AC magnetic field generator 250. 

[0029] According to the changes in the AC magnetic field sensed by the magnetic sensors 220, a position/attitude 

IS measuring section 5000 (not shown in FIG. 1) measures the positions/attitudes of the magnetic sensors 220 in the 
sensor coordinate system 8010 defined around the AC magnetic field generator 250. Position/attitude transfomriation 
sections 5060 (5060L and 5060R) (not shown in FIG. 1 ) transfom the positions/attitudes of the magnetic sensors 220 
In the sensor coordinate system 8010 into the positions/attitudes of the players' viewpoints In the global coordinate 
system 8000, based on the calibration data (acquired by a calibration technique described later) stored in calibration 

20 data storage sections 5080 (5080L and 5080R) (not shown In FIG. 1). Incidentally, as the magnetic sensors 220. AC 
magnetic field generator 250, and position/attitude measuring sections 5000, this embodiment employs position/atti- 
tude measuring system Fastrak from Polhemus Inc. Besides, the position/attitude transfomnation sections 5060 and 
calibration data storage sections 5080 are implemented, for example, using an Onyx2 computer system from SGI. 
[0030] When the player lowers his/her gaze obliquely to look at the surface of the table 1000, the changes In the 

25 position/attitude of the player's viewpoint are detected by the magnetic sensor 220. The surface of the table 1000, 
virtual puck 1500 described above, real mallets 260 (260L and 260 R), and virtual goals 1200 (1200L and 1200R) are 
shown through the HMD 210, based on the detected changes In the position/attitude. In this way, the changes In the 
position/attitude of the player's viewpoint is always detected by the magnette sensor 220 and the player can see the 
picture and real space, based on the detected position/attitude. 

30 [0031 ] The position and attitude of an object B In a coordinate system A will be represented herein by a 4 x 4 matrix 
Mqa. In other words, It is the matrix for coordinate transfomnation from the coordinate system A Into the coordinate 
system B defined by the object B and defines a transfomnation Pq = Mq^ Pa ^or changing coordinates Pa = (X^, Ya, 
Za. 1)T In the coordinate system A into coordinates Pg = (Xg, Yb, Zb. 1 V in the coordinate system B. To put It another 
way, the position/attitude Mwv of the viewpoint 1901 of the player 2000 In the global coordinate system 8000 is the 

35 coordinate transfomnation matrix (Py = Myw Pw) changing coordinates P^ = (Xw. Yw Zyv- ^)'^ the global coor- 
dinate system 8000 into coordinates Py = (Xy. Yy, Zy, 1)T In the coordinate system of the player's viewpoint. 
[0032] The transformation matrix M that represents the position/attitude of the object Is given by the product of a 
rotation matrix Rx around the X axis, rotation matrix Ry around the Y axis, rotation matrix R^ around the Z axis, and 
translation matrix T (all are 4 x 4). The relationship M = RT = Rz Rx Ry T holds among these matrices, which are give by: 



40 
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[Equation 2] 
Rxtex) = 



[Equation 3] 



Ry(ey) 



10 0 0 

0 COS 6x sin 6x 0 

0 - sin Ox cos Ox 0 

0 0 0 1 



cos Gy 0 - sin By 0 

0 10 0 

Sin 6y 0 cos Oy 0 

0 0 0 1 



[Equation 4] 



R2(ez) = 



cos 8z sin 6z 0 0 

- sin 8z cos 6z 0 0 

0 0 10 

0 0 0 1 



where x. y, z, ex, ey, and ez represent the position/attitude of the object. More particularty, x, y, z represent the 

position, ex the elevation angle, ey the direction angle, and ez the slope. 

[0033] The following describes with reference to FIG. 19 how the position/attitude transformation section 5060L 
determines the position/attitude Myw of the viewpoint 1901 of the player 2000 in the global coordinate system 8000. 
[0034] In FIG. 19, the position/attitude of the magnetic field generator 250 in the global coordinate system 8000 
(equivalent to the position/attitude of the sensor coordinate system 8010 In the global coordinate system 8000) Is 
denoted by M-rw. position/attitude of the magnetic sensor 220L in the sensor coordinate system 8010 Is denoted by 
MsT. and relative position/attitude of the viewpoint 1 901 of the player 2000 as viewed from the magnetic sensor 220L 
is denoted by Mys- 

[0035] Then the position/attitude M^v the viewpoint 1 901 of the player 2000 in the global coordinate system 8000 
Is given by: 



Myw = Mvs • '^ST * "^TO 



(Equation A) 



[0036] Of these positions/attitudes, Mst is the Input into the position/attitude transf omiatlon section 5060L, Myw 
the output from the position/attitude transfomiation section 5060L, and Mys and are the calibration data needed 
to convert Mgj into Myw- "That is, the position/attitude transfomiation section 5060L calculates Myw according to Equa- 
tion A using the input Msj from the position/attitude measuring section 5000 as well as Mvs and M^vv stored in the 
calibration data storage section 5080L described later and outputs it to the image generation section 5060L described 
later. 

<Configuration of equipment during calibration> 

[0037] FIG. 20 shows the equipment configuration used for the calibration of the game device according to this 
embodiment. As shown in FIG. 20, the calibration uses almost the same equipment configuration as that for game 
playing. Specifically, the configuration includes the AC magnetic field generator 250, magnetic sensor 220L, position/ 
attitude measuring section 5000, and HMD 21 OL as Is the case with the configuration for game playing, but it has a 
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calibration section 5055L and operating section 5090 Instead of the position/attitude transfomriatlon section 6060L and 

the image generation section 5050L. 

[0038] FIG. 4 shows an outline of calibration setup. To perform calibration, the position/attitude Myw viewpoint 
1901 of the player wearing the HMD 21 OL is moved to a predetermined position/attitude (or the HMD 21 OL is 

5 moved by an operator (not shown) to such a position/attitude that the viewpoint 1 901 of the player will be placed in the 
position/attitude M^yw when the player puts on the HMD 21 OL) and the output M^st ^^m the position/attitude measuring 
section 5000 Is acquired at that time. Using the operating section 5090L, the player or unshown operator enters the 
information that the viewpoint 1901 is placed in the position/attitude hA^^^, According to the Input Into the operating 
section 5090L, the calibration section 5055L receives the output from the position/attitude measuring section 

10 5000 at that time (I.e., at the time when the viewpoint 1901 Is placed in the position/attitude M^yw)- calculates the 
calibration data Mys or M^, whichever Is unknown, based on the position/attitude M^wv ^"^ position/attitude M^sj, 
and outputs the results to the calibration data storage section 5080L. 

[0039] With this embodiment, It is assumed that the matrix Mjyf for coordinate transfonmatlon from the global coor- 
dinate system into the sensor coordinate system has been derived by some technique (e.g. , a method described later) 

15 and that unlcnown calibration data is only Mys- The calibration data storage section 5080L stores the calibration data 
Mys received from the calibration section 5055L and stores the M^w as l<nown values. The transfomnation matrices 
Mys and M^ are referenced by the position/attitude transformation section 5060L during game playing. 
[0040] A calibration grid 6000 can be placed at a fixed position in real space and has on itself an established position 
that will serve as a guide when moving the HMD 2 1 0L. The fixed position and established position have been designed 

20 such that when the calibration grid 6000 is placed at the fixed position and the HMD 21 OL is moved to the established 
position, the viewpoint 1901 of the player who wears the HMD 21 OL will be placed near the position/attitude MOyw- 
The calibration grid 6000 is built of plastic pipes to prevent interference when the magnetic sensor 220L receives the 
AC magnetic field outputted by the AC magnetic field generator 250 and helps the viewpoint 1901 remain in the position/ 

attitude MOyyy. 

25 [0041 1 Markers 1 600 (1 600a, 1 600b, . . .) that have known values In the global coordinate system are placed on the 
table 1000. They are used in the processing by the calibration section 5055L (described later) to help the viewpoint 
1901 remain in the position/attitude MOyw- Incidentally, this embodiment uses four markers 1600, but more markers 
may be used. 

[0042] Now the processing in the calibration section 5055L will be described in detail. FIG. 5 shows detailed config- 
30 uration of the calibration section 5055L according to this embodiment. The calibration section 5055L is Implemented, 

for example, using an Onyx2 computer system. 

[0043] As shown in FIG. 6, the calibration section 5055L is composed of a memory 5046L, virtual marker picture 
generation section 5075L, marker location storage section 5035L, and calibration data calculation section 5040L. 
[0044] The marker location storage section 5035L stores the locations of the markers 1 600 in the global coordinate 

35 system and outputs the marker locations to the virtual marker picture generation section 5076L. 
[0045] The memory 5045L stores the predetennined position/attitude M^yy^ of the viewpoint 1 901 . 
[0046] Based on the marker locations stored In the marker location storage section 5035L and the position/attitude 
MOywOf the viewpoint 1901 stored in the memory 5045L, the virtual marker picture generation section 6075L calculates 
the location of each marker on the display screen where the marker will be seen by the player If the viewpoint 1 901 is 

40 in the position/attitude MOyw, generates a virtual picture representing an X mark at that location, and makes it displayed 
on the display section 21 OL 

[0047] The display section 21 OL displays the virtual pictures of the markers generated by the virtual marker picture 

generation section 5075L while transmitting the markers 1600 in real space optically through Itself. 

If the viewpoint 1901 is located in the position/attitude M^yw, the marker images in real space and the virtual pictures 

45 of the markers as viewed by the player should match. 

[0048] The operating section 5090L detects that the viewpoint 1901 is placed in the position/attitude M^yy^, based 
on the Infonnatlon entered by the player or unshown operator as described above, and sends an instruction to calculate 
calibration data to the calibration data calculation section 5040L. By rotation and translation of HMD 21 OL (and thus 
the viewpoint 1 901 ), the player superimposes the virtual pictures of the markers displayed in the display section 21 OL 

so and the marker Images in the real space optically transmitted through the display section 21 OL, and when they are 
overlaid sufficiently, he/she enters Input through the operating section 5090L, for example, by pressing a specific key. 
[0049] /According to the instructions sent from the operating section 5090L, the calibration data calculation section 
5040L receives the output M^gi- from the position/attitude measuring section 5000 at the time when the instructions 
are entered (I.e., at the time when the viewpoint 1901 is placed in the position/attitude M^yw) as well as the position/ 

55 attitude M^yw stored In the memory 5045L and Mjw stored in the calibration data storage section 5080L. 

[0050] From Equation A, it follows that the following relationship exists among the data processed in the calibration 
data calculation section 5040L: 
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M°vw = Mvs • ' ^VN (Equation B) 

[0051] Now rearranging Equation B, 

5 

Mvs = l^^vw • "^Tw"^ • (Equation C) 

[0052] The calibration data calculation section 5040 L calculates the unknown calibration data Mys according to Equa- 
tlon C and outputs the results to the calibration data storage section 5080L. 

[0053] FIG. 6 shows a process flowchart of the calibration apparatus according to this embodiment. 

The program code according to this flowchart is stored in unshown memory such as a RAM or ROM In the apparatus 

of this embodiment, and read out and executed by a CPU (not shown). 

[0054] In Step S1 040, the virtual marker picture generation section 5075L generates virtual pictures in the manner 
^5 described above, and makes it displayed in the display section 21 OL. The image displayed in the display section 21 OL 
in Step SI 040 is shown in FIG. 7A. Each X mark in the figure represents the virtual picture of each marker. The markers 
1 600 placed in real space is viewed by the player optically through the display section 21 OL. The image viewed by the 
player in the display section 21 OL is shown in FIG. 7B. 

[0055] In Step S1 070. the virtual pictures of the markers and the marker images in real space are overlaid exactly 
20 by rotation or translation of HMD 21 OL (and thus the viewpoint 1901). (The state shown in FIG. 7C.) 
[0056] When they are overlaid exactly, the operating section 5090L is manipulated in Step SI 080. 
[0057] In Step S1 090, the calibration data calculation section 5040L calculates the coordinate transformation matrix 
Mvs in the manner described above and outputs the results to the calibration data storage section 5080L. 

2^ <Derivatlon of the position/attitude Wij^/ of the sensor coordinate system 8010 in the global coordinate system> 

[0058] The following describes how to find the position/attitude M-^vv oi the sensor coordinate system 801 0 In the 
global coordinate system 8000. 

[0059] The calibration grid 6000 is placed at a fixed position in real space and the magnetic sensor 220L is placed 

^ at a specified position (point of measurement) on the calibration grid 6000. There is more than one specified position. 
According to this embodiment, the output of the magnetic sensor 220L is measured at three points on the calibration 
grid 6000. The results of the measurements are entered in to the calibration section 5055L. 
[0060] The calibration section 5055L calculates the position/attitude M-nv of the sensor coordinate system 8010 in 
the global coordinate system 8000, based on the three measurements taken with the magnetk; sensor 220L. and stores 

^ the results in the calibration data storage section 5080. 

[0061] The following describes in detail how the calibration section 5055L finds the position/attitude Mrw. 
[0062] FIG. 18 is a conceptual diagram illustrating how the magnetic sensor measures its position/attitude. Let 
denote the origin of the global coordinate system; X^. Y^, and the x, y, and z axes of the global coordinate 
system, respectively; Os - the origin of the sensor coordinate system (expressed as coordinates in the global coordinate 

^ system); Xs, Ys. and Zg - the x, y, and z axes of the sensor coordinate system (expressed as unit direction vectors in 
the global coordinate system) ; Pq, Pz. Pzx " ^^^^^ points of measurement described above, respectively (expressed 
as coordinates in the global coordinate system); Vq, V^, Vzx - the three points of measurement described above, 
respectively (expressed as coordinates in the sensor coordinate system). The three points described above must 
satisfy the following conditions. In other words, the fixed position for the calibration grid 6000 and the three points of 

^ measurement on the grid must be set such that the following conditions will be satisfied. 
[0063] • The position of P© in the global coordinate system must be known. 
[0064] • Pz must be such that the vector (Pz - Pq) parallel to, and in the same direction as, 2^. 
[0065] • Pzx niust be such that the plane containing the three points Pq. Pz. and Pzx is parallel to the z-x plane in 
the global coordinate system and that the x component of the vector (Pxz - Pq) 'n the global coordinate system will be 

^ positive. 

[0066] Based on these conditions, the position/attitude M™ of the sensor coordinate system 801 0 is detemnined in 
the following order. 

[0067] First, since the vector (Pz - Pq) is parallel to, and in the same direction as, Z^, the unit vector Zs in the z 
direction of the sensor coordinate system is determined according to the following equation: 

55 

Zs = (Vz-Voy"Vz-Vol 



8 



EP 1213 686 A2 



[0068] Next, since the vector (Pzx • Pq) 'S located on the z-x plane of the global coordinate system and has a positive 
X component, the unit vector Ys in the y direction of the sensor coordinate system is detemiined according to the 
following equation: 

^ Ys = (Zs X {Vzx-Vo))/l2s X (Vzx-Vq)" 

[00691 Next, using Zs and Ys above, the unit vector Xs in the x direction of the sensor coordinate system is determined 
according to the following equation: 

10 

Xs = Ys X Zs 

[0070] From Zg, Yg, and Xs thus obtained, the attitude of the sensor coordinate system in the global coordinate 
15 system can be determined as follows. Let Xg = (Xx, Yx. Zx). Ys = (Xy, Yy, Zy), and Zs = (X2, Yz, Z^), and the attitude 
R^w of the sensor coordinate system in the global coordinate system is given by: 



[Equation 5] 




0001 



[0071] Next, the position of the sensor coordinate system in the global coordinate system (i.e., the position Og of 
the origin of the sensor coordinate system in the global coordinate system) is detemiined. The following method uses 
30 Pq, but P2 or may be used if their global coordinates are known: 

35 [0072] Hence, the translation matrix T-rw that represents the position of the sensor coordinate system in the global 

coordinate system can be determined according to Equation 1 . 

[0073] Consequently, the position/attitude M-pw of the sensor coordinate system In the global coordinate system can 
be derived using the following equation. 

^ Mjyy = RyW^TW 

[0074] Based on the signals from the magnetic sensor 220L that tal<es measurements at multiple points of meas- 
urement, the position/attitude Myw of the sensor coordinate system in the global coordinate system can be determined 
45 by the above method. 

[0075] Incidentally, the position/attitude M-iy, of the sensor coordinate system in the global coordinate system may 
be detemnined by trial and eror using measurements taken with a measuring tape, protractor, etc. Also, needless to 
say, the position/attitude of the AC magnetic field generator may be determined using any available measuring means. 

50 <3D Image generation/presentation system> 

[0076] FIG. 8 is a drawing showing the configuration of a 30 image generation/presentation system that utilizes the 
calibration data obtained through the abovennentioned calibration on the game device of FIG. 1. 
[0077] The image generation/presentation system outputs respective 3D virtual pictures (puck 1 500 and goals 1200 
55 in FIG. 2) to the HMD 21 OL of the player 2000 and HMD 21 OR of the player 3000. The 3D virtual pictures for the players 
2000 and 3000 are generated by the image generation sections 5050L and 5050R, respectively. This embodiment 
uses an Onyx2 computer system from a US company SGI for each of the image generation sections 5050L and 5050R. 
[0078] The image generation section 5050L (5050R) receives puck position information and the like generated by a 
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game status control section 5030 and the position/attitude of the viewpoint of the player 2000 (3000) output from the 
position/attitude transformation section 5060L {5060R) and generates images to be displayed on the HMD 21 OL (21 OR). 
The game status control section 5030 Is Implemented using the ONYX2 computer system. 
[0079] A CCD camera 230 (also shown in FIG. 1 ) fixed above the center of the table 1 000 covers the entire surface 

5 of the table 1000 by its field of view. The mallet Information Including the mallet position information acquired by the 
camera 230 is entered Into a mattet position measuring section 501 0, which similarly is Implemented using the ON YX2 
computer system from SGI. The mallet position measuring section 5010 detects the positions of the mallets, i.e., the 
positions of the hands, of the two players. The information about the hand positions are entered into the game status 
control section 5030. In short, eveiything about the status and progress of the game is determined by the positions of 

10 the mallets. 

<Maiiet position measurement> 

[0080] FIG. 9 to FIG. 11 are flowcharts showing the control procedures for measurement of mallet position. 
IS [0081] In the air hockey game, the player will never put his/her mallet in the other player's area. Therefore, to search 
for the mallet 260L (260 R) of the left player 2000 (right player 3000), the search process can be concentrated on the 
Image data IL in the left field (image data IR) as shown In FIG. 11 . The image captured by the CCD camera 230 at a 
fixed position can be easily split in two as shown in FIG. 12. 

[0082] In the flowchart of FIG. 9, the mallet 260L of player #1 (player 2000) is searched for in Step S100 and the 
20 mallet 260 R of player #2 (3000) is searched for In Step S200. Therefore, only the search operation (Step S200) for 
the mallet of the right player will be described for convenience. 

[0083] In Step S210, the above-mentioned image (multivalued Image) captured by the imaging section 230 is ac- 
quired. In Step S21 2, subroutine "Local Search" is run on the image data IR about the right half of the captured Image. 
Details are shown in FIG. 10. If the coordinates (x, y) of the mallet position in an Image coordinate system Is found In 
2S Step S212, the flow goes from Step S214 to Step S220, where the coordinates (x, y) of the mallet position in the Image 
coordinate system is transformed Into the coordinate position (x*, y') in the coordinate system (see FIG. 14) of the table 
1000 according to Equation 6. 

30 (Equation 6] 



hx'' 






hy' 




y 


h 




1 



35 

where hAj is a known 3x3 transformation matrix for the calibration of the image coordinate system and table 
coordinate system. The coordinate position (x', y*) obtained In Step S220 is sent to the game status control section 5030. 
[0084] If the mallet cannot be found In the local area, "Global Search" Is run in Step S21 6. if the mallet is found by 

40 the "Global Search," the coordinate position is transformed Into the position in table coordinate system In Step S220. 
The coordinate position retrieved by the local or global search Is used for a local search for the mallet In the next process. 
[0085] FIG. 10. shows the process of a local search for the mallet (i.e., details of Step S212). Although the process 
shows a search in the right field for convenience, it can similariy be applied to search in the left field. 
[0086] In Step S220. the rectangular area made of (2A + 1 ) x (2B + 1 ) picture elements and defined by the following 

45 equations is extracted. 

x = II'x-A, I'x + AJ 



y = ll'y-A, I'y + B] 

where i'x and IV are any coordinate values in the search area IR. and A and B are constants that detemnlne the 
size of the search area. Such a search area will look like the one shown In FIG. 13. 
55 [0087] Step S230 is the process of extracting the picture elements whose character evaluation value IS (x, y) satisfies 
specific conditions from all the picture elements (x, y) in the rectangular area defined in Step S220. In this process, it 
is preferable to use a methods that extracts the picture elements whk:h are similar to the intensity value of the infrared 
rays emitted from the Infrared emitter of the mallet. 
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[0088] In other words, In Step S232, any picture element whose similarity IS exceeds a designated threshold Is 
searched for. If such picture elements are found, the cumulative value of their generation rate Is stored on a counter 
N. Also, the X cooixJinates and y coordinates of these picture elements are accumulated in registers SUMx and SUMy. 
Thus, 



10 



N = N + 1 



SUMx = SUMx -i-x 



SUMy = SUMy -i-y 



IS [0089] When Step 8230 is completed, the number N of all the picture elements in the area of FIG. 13 which are 
similar to the pattem of the infrared rays emitted from the mallet as well as cumulative values SUMx and SUMy of the 
coordinates are obtained. If N = 0. the result "Not Found" is output in Step S236. If N > 0. something that looked like 
a mallet was found and the position (Ix, ly) of the mallet is calculated according to the following equations in Step S238: 



20 



25 



30 



35 



Ix = SUMx/n 



ly = SUMy/n 



[0090] The mallet position (Ix, ly) is transformed into coordinates in the table coordinate system. 
[0091] FIG. 1 1 shows detailed procedures for the global search perfomied in Step S21 6. In Step S240 shown in FIG. 
11 , among the picture elements contained in the image IR in the right field, the maximum value of the character eval- 
uation values IS of the picture elements that satisfy the following conditions is stored in a register Max. 

{(X, y)l X > 0, X < Width, x = nC, 
y > 0, y < Height, y = mD 



(where n and m are integers)} 

where C and D are constants that detenmlne the roughness of the search. The definitions of width and height 
are shown In FIG. 1 6. In short, it is judged in Step S242 whether the character rating IS exceeds the threshold stored 
in the register Max. If any appropriate picture element is found, the character rating is set as a new threshold in Step 
40 S244 as follows: 

Max = IS (X, y) 



45 



50 



55 



lx = x 



ly = y 



[0092] In Step S246, the coordinates (Ix, ly) of the most probable picture element found In the global search is passed 
to Step S220. 

[0093] In this way, the mallet Is found In the Image, its coordinates are transfomned into the coordinates In the table 
coordinate system, and the results are passed to the game status control section 5030. 

<Game status control> 

[0094] FIG. 14 shows the game field for air hockey games according to this embodiment. This field is defined In the 
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two-dimensional plane on the table 1 000 and has x and y axes. It also has two - left and right - virtual goal lines 1 200L 
and 1 200R as well as virtual walls 1 300a and 1 300b installed on top and bottom In FIG, 1 4. The virtual goal lines 1 200L 
and 1200R and virtual walls 1300a and 1300b have known coordinates and are not moved. In this field, the virtual 
image o( the pucic 1 500 moves in accordance with the movement of the mallets 260R and 260L. 
5 [0095] The puck 1 500 has coordinate infomiation Pp and velocity Infonnatlon Vp of the current position, the left mallet 
260L has coordinate infonnation Psl and velocity Information Vsl of the cunrent position, and the right mallet 260R 
has coordinate Information Psr and velocity Infomnation Vsr of the curent position. 

[0096] FIG. 15 is a flowchart Illustrating the processing procedures carried out in the game status control section 
5030. In Step S10. the initial position Ppo and Initial velocity Vpo of the puck 1500 are set. 
10 [0097] The puck 1500 perfomis unlfomi motion at velocity Vp. If it hits a mallet or wall, a perfectly elastte collision 
takes place. That Is, the speed remains unchanged and the velocity direction is reversed. 

[0098] The game status control section 5030 obtains velocity inf omiation from the position infomriation Pg of each 
mallet calculated by the mallet position measuring section 5010. 

[0099] Step SI 2 is repeated at time Intervals of At until the outcome of the game Is decided (either player scores 
15 three points first in Step S50). 

[0100] Then the position of the puck 1 500 Is updated to 

Pp = Ppo+VpoAt 

20 

[0101 J The position of the puck 1500 after the Inttlal position and velocity setting are generally given by 

Pp=Pp + Vp.At 

25 

[0102] In Step S14, It is checked whether the updated puck position Pp is in the field of player #1 (left player). The 
case in which the puck 1500 is located on the side of the left player will be described below. 
[0103] In Step SI 6. it is checked whether the puck is currently located such that it interferes with the mallet 260L of 
the left player. If the puck 1500 interferes with the mallet260L, it means that the left player 2000 manipulated the mallet 
30 in such a way that the mallet 260L collided with the puck. Thus, to reverse the motion of the puck 1 500. the sign of the 
X direction component of the puck's (1 500) velocity Vp is changed in Step SI 8. Then the flow advances to Step S20. 
[0104] Incidentally, instead of simply changing the sign of the x direction component of the velocity Vp, the puck may 
be made to advance In the opposite direction with the manipulation velocity of the mallet added as follows: 

^ Pp--Ppx + Vsu< 

[0105] On the other hand, if the puck currently does not interfere with the mallet 260L of the left player ("NO" in Step 

SI 6), the flow advances directly to Step S20. 
40 [0106] In Step S20, it Is checked whether the puck is currently located such that it collides with the virtual wall 1 300a 

or 1 300b. If the answer in Step S20 is YES, the y direction component of the puck's velocity Vp is reversed in Step S22. 

[01 07] In Step S24, it is checked whether the puck is currently located to the left of the goal line of the left player. If 

YES, the opposition player, i.e., the right (#2) player gets a score in Step S26. In Step 850, it is checked whether any 

of the players scored three points or more first. If YES, the game finishes. 
45 [0108] If it tums out in Step S14 that the puck position Pp is in the field of the right player (player #2), Step S30 and 

subsequent steps are performed. Steps S30 to S40 are practically the same as Steps SI 6 to S26. 

[0109] So the progress of the game is managed in the manner described above. The progress of the game, whteh 

is represented by the positions of the puck and mallet, is entered in the image generation sections 5050 (5050L and 

5050R). 

50 [0110] FIG. 17 shows a flowchart of processes carried out in the position/attitude transformation sections 5060 
{5060L, 5060R) and image generation sections 5050 (6050L, 5050R). The processes in this flowchart are common to 
the left and right players. 

[0111] The position/attitude transformation section 5060L receives the position/attitude M^r of the magnetic sensor 
in the sensor coordinate system from the position/attitude measuring section 5000 in Step S1 701 . Then, in Step SI 702, 
55 it reads the calibration data Ujyj and My^^ from the calibration data storage section 5080. In Step S1703. it determines 
the position/attitude of the player's viewpoint 1901 using Equation A. 

[0112] The image generation sections 5050 read the position and shape data of the goals and puck from memory 
(not shown) in the game status control section 5030 in Step SI 704. generate pictures of the goals and puck based on 
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the position/attitude Mvw the player's viewpoint 1901 in Step S1705. and output the generated pictures to the HMD 
21 OL and HMD 210R of the respective piayers In Step S1706. 

[01 131 As described above, the calibration apparatus and method thereof according to this embodiment can calculate 
the positions/attitudes of the players' viewpoints even if the calibration data needed to convert the output values of the 
5 magnetic sensors into the players' viewpoints are unknown. 

[Second Embodiment] 

[0114] In the first embodiment, the relative positions/attitudes Mys of the players' viewpoints as viewed from the 
10 magnetic sensors are unknown. In other words, both the translation component Tys and rotation component Rys of 
the transfomnation matrix Myg are unknown. 

[0115] Also, in the first embodiment, both the translation component T-rw and rotation component Rxw of the position/ 
attitude M-rvv of the sensor coordinate system in the global coordinate system are known. 

[0116] In relation to the second embodiment, the following describes how to calculate unknown calibration data, i. 
15 e., the rotation component Rvs of Mys and translation component T-r^ of M-rw when the translation component Tvs 
of Mvs and rotation component Bjy^ot Mj^ are known in some way or other. This embodiment has the same equipment 
configuration as that of the first embodiment, and differs only In the processing inside the calibration data calculation 
section 5040L. 

[01 17] The translation component Tys of Mys (i o-. the relationship between the position of the magnetic sensor and 
20 position of the user's viewpoint) is measured directly, for example, with a rule. The rotation component Ryw of (Vljw is 
calculated using, for example, the method of finding Ryw described In relation to the first embodiment. 
[0118] Alternatively, these values may be derived by trial and error as described in relation to for the first em- 
bodiment, or any other measuring technique may be used. 

[0119] It is assumed that the known parameters are stored in the calibration data storage sections 5080 as is the 
25 case with the first embodiment. 

[0120] Separating Equation B into rotation and translation components, 

R^vwT'vw = RvsTvs • m'st • fWT™ t^^^^^'^" °) 



30 



35 



40 



45 



Re-arranging Equation D, 



Rvs ' R°vwT^vwTtw"' = TvsM'stR™ ^) 



[0121] Since all the terms on the right side have known values, if their product Is calculated and replaced with a 
matrix M' and the matrix is decomposed Into M' = R' T (where R' is the rotation component and T is the translation 
component), then Equation E is written as 

Rvs'R°vwT°vwW=R'T' (Equation F) 

[0122] Since the rotation component and translation component exist as identities in the right side member and left 
side member of Equation F, respectively, Rys is given by: 

Rys = R^vwR* ^ 



[0123] Similariy, T-rw is given by: 

50 

'tw- ' ' VW 

[0124] Finally, the resulting calibration data Rys and T-^w are output to the calibration data storage section 5080. 

55 
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[Third Embodiment) 

[0125] This embodiment concerns the method of calculating unknown calibration data, i.e., the position/attitude M-pv 
of the sensor coordinate system in the global coordinate system when the relative positions/attitudes of the players' 
5 viewpoints Mys as viewed from the magnetic sensors are known in some way or other. 

[0126] This embodiment has the same equipment configuration as that of the first embodiment, and differs only In 
the processing inside the calibration data calculation section 5040L. The unknown calibration data Mtw 's calculated 
by rean-anging Equation B as follows, and is output to the calibration data storage section 5080L. 

= M°sT * ^vs ^ " ^°vw 

[0127] Alternatively, Mys may be derived by the method of the first embodiment; by trial and error based on values 
measured with a rule, protractor, etc.; or by any other measuring means. 

15 

[Fourth Embodiment] 

[0128] Although the first to third embodiments relate to calibration of magnetic sensors which are position/attitude 
sensors for measuring the positions/attitudes of ob]ects, the present Invention can also be applied to calibration of 

20 attitude sensors that measure only the attitudes of objects. The fourth embodiment concerns the method of detemrilning 
the calibration data needed to convert the output values of attitude sensors (i.e., the attitudes of the attitude sensors 
in the sensor coordinate system) into attitudes of the players' viewpoints in the global coordinate system for a game 
device that uses attitude sensors. This embodiment has the same equipment configuration as that of the first embod- 
iment except that it comprises attitude sensors and an attitude measuring section. 

25 [01 29] Now let 1^ denote the attitude of the sensor coordinate system In the global coordinate system , Rgr " the 
attitude of the attitude sensor (output values from attitude sensor) in the sensor coordinate system, and Rvs - the 
relative attitude of the player's viewpoint as viewed from the attitude sensor. Incidentally, although most attitude sensors 
do not have expltoit devices for defining a sensor coordinate system, such as the AC magnetic field generator in the 
case of magnetic sensors, if it is assumed that the sensor attitude in the global coordinate system corresponds to R-pvy 

30 when sensor output is pointing to the origin (i.e., when Rsr = "s output), the coordinate transfonnatlon is given by a 
relation similar to those used in the first to third embodiments. Thus, the Equation A can be expressed as 

Rvw=Rvs-Rst-R™ (Equation G) 

35 

where Rgj is the input Into the position/attitude transfomiation section 5060L, Ryw is the output from the position/ 
attitude transfomiation section 5060L, and Rys and R-f^v are the calibration data needed to convert Rst into Ryw- Thus, 
the position/attitude transformation section 5060L calculates Ryw according to Equation G using the input Rgy trom 
the position/attitude measuring section 6000 as well as Rys and Rn^ stored in the calibration data storage section 

^ 5080L. The position/attitude transfonmation section 5060L further calculates the position/attitude Myw of the player's 
viewpoint in the global coordinate system, based on the calculated attitude Ryw of the player's viewpoint in the global 
coordinate system and the position Tyw o1 the player's viewpoint in the global coordinate system, and outputs the 
results to the image generation section 5050L. (Twv may be the values measured by a position sensor such as an 
ultrasonic sensor or optical sensor Alternatively, predetemiined fixed values may be used if the position of the viewpoint 

^5 does hot move or the amount of travel is negligible. Also, position information obtained by any other means may be 
used.) 

[0130] This embodiment assumes that of the attitude R-rw of the sensor coordinate system, only the rotation com- 
ponent Ry-rw around the y axis (i.e., in the azimuth direction) is unknown and that the rotation component Rx^w around 
the X axis and rotation component Rz-pw around the z axis are unit matrices (i.e., Rrv/ = RVtw)- Since many attitude 
^ sensors can output values in a coordinate system in real space by measuring the direction of the earth gravitation 
except values in the azimuth direction (around the y axis, in the yaw direction), the y axes of the global coordinate 
system and sensor coordinate system coincide if the global coordinate system is defined as being parallel to the ground 
surface. Thus, Rx-pvv anci Rz-p/v can be considered to be unit matrices. 

[01 31 ] This embodiment also assumes that, of the relative attitude Rvs of the player's viewpoint as viewed from the 
55 sensor, the rotation component Rysv around the y axis of its inverse matrix Rsv (relative attitude of the attitude sensor 
as viewed from the player's viewpoint) is known in some way or other while the rotation component Rxsv around the 
X axis and rotation component Rzqv around the z axis are unknown. Rysv may be derived by trial and error using 
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values measured with a protractor, etc., or by any other measuring means. 

[0132] The Icnown data Rxjvv. Rztw. and Rysv ^^ave been stored in the calibration data storage section 5080L. 
[0133] Calibration is perfomned by moving the attitude Ryw oi the viewpoint 1901 of the player wearing the HMD 
21 OL to a predetemiined attitude ROyyv (= fV^vw) and acquiring the sensor output R^ a^ ^at time. Using Equation 
5 G, the relationship among these data can be expressed as 

Ry^'vw = (RzsvRxsvRysv) ''rVRVtw (Equation H) 

10 Rearranging Equation H 

RzsvRxsvRYsvRy^vw = ^^sr^^sr^^sr^yvN 

15 [0134] Since both left-side and right-side members are products of rotation components around the z, x, and y axes, 
an identity holds for the rotation component around each axis: the z, x, and y axes. The identities of the rotation 
components around the z and x axes are given as 

RXgy = Rj^^ST 

25 from which Rzsv and Rxsv can be detemnined. 

[0135] On the other hand, the identity of the rotation component around the y axis can be determined from the 
following equation. 

30 RysvRy°vw = Ry°srRyTO 

[0136] Thus, H^jYf is given as 

35 RyTw = RysvRy**vwRyV^ 

[0137] The calibration data calculation section 5040L according to this embodiment calculates the calibration data 
RZsv. R^sv. and Ry-fw as described above, further calculates Rvs {= (Rzsv R^sv Kysv)"l) and R^vr (= RYtw)* and 
then outputs the results to the calibration data storage section 5080L. 
40 [0138] The memory 5045L according to this embodiment retains the predetennlned position T^yw viewpoint 
needed for the virtual mariner picture generation section 5075L to generate virtual pictures of maricers In addition to 
the predetemiined attitude R%vv viewpoint needed to calculate the calibration data described above. 

<Variation 1> 

45 

[01 39] Although the embodiments described above use an optical HMD, the present invention Is not limited to optical 
HMDs, but It can also be applied to video see-through type HMDs. 

[0140] In that case, the measuring object, which is the player's viewpoint 1901 according to the embodiments de- 
scribed above, will be an imaging apparatus (e.g., a video camera) installed in the video see-through type HMD. The 
so predetermined position/attitude M^vw will be given as the position/attitude of the viewpoint of the imaging apparatus 
and the transtomriatlon matrix Mvs will be given as the transfonnation matrix that represents the position/attitude of 
the imaging apparatus as viewed from the magnetic sensor 220L. 

[0141] According to this variation, the virtual mariner picture generation section 5075L superimposes a virtual picture 
of mariners over the picture (marlcer image) of real space captured by the Imaging apparatus, based on the predeter- 
55 mined position/attitude M^vw. and displays the resulting picture in the display section. The markers 1 600 must be such 
that they can be distinguished from the table 1 000 In the marker image. The operator manipulates the position/attitude 
of the imaging section to overiay the real and virtual pictures of the mari<ers. In video see-through mode, the display 
section does not always need to use the HMD 210L of the game device during calibration unlike in optteai see-through 
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mode, but a CRT display or other display units may be used. 
<Variatlon 2> 

5 [0142] Even if a game device uses an optical HMD, calibration may be performed in video see-through mode as with 
Variation 1 if the HMD 21 OL is equipped with an imaging apparatus whose relative position/attitude as viewed from the 
player's viewpoint is known. 

[01 43] According to this variation, the virtual marl<er picture generation section 5075L superimposes a virtual picture 
of marlcers over the picture (marker image) of real space captured by the Imaging apparatus, according to the position/ 
10 attitude of the imaging apparatus in the global coordinate system, determined by a predetennlned position/attitude 
M^vw player's viewpoint and the position/attitude of the imaging apparatus as viewed from the player's viewpoint^ 
and displays the resulting picture In the display section. If the HMD 21 OL is used as the display section, It is desirable 
to make display surface light-tight by adjusting its transmittance. 

15 <Variatlon 3> 

[0144] Although the above embodiments use the markers 1600 whteh are point infomiatlon - as a guide in align- 
ment, any geometry infomnation may be used as long as It provide visual clues when moving the position/attitude of a 
viewpoint to a predetermined position/attitude M^wv- example, the user may capture the geometry of a real object 

20 such as a desk from the environment and draw Its wire-frame model according to a predetemnlned position/attitude 
M^vw' ^^^'s case, it Is possible to lead the viewpoint to the position/attitude M^vw moving the viewpoint so as to 
match the image of the real desk and the superimposed picture of the wire-frame model. The viewpoint can also be 
led to the position/attitude M^vw follows: If the virtual object (e.g., the virtual puck in the first embodiment) displayed 
after calibration is drawn according to the predetennlned position/attitude MOyyy, the viewpoint can be moved so as to 

25 place the virtual puck at the appropriate position on the real desk, incidentally, it goes without saying that some of the 
above-mentioned approaches can be used In combination. 

<Variation 4> 

[0145] Although the above embodiments use the calibration grid 6000 to help place the viewpoint in the position/ 
attitude M^vw' i^^t always necessary to use a calibration grid: visual information alone can be used as a guide. It 
is also possible to place the viewpoint in the position/attitude M^wv placing the HMD 21 OL in an established attitude 
at an established position on the calibration grid 6000 instead of using visual information. Or any other method may 
be used as long as it leads the viewpoint to a predetennlned position/attitude (or attitude). 

<Varlation 5> 

[0146] Although the above embodiments have been applied to air hockey games, It goes without saying that they 
can be applied to systems that present mixed reality other than air hockey games. Besides, it Is not that they are 
suitable only for coordinated wori( by two persons. They are also applicable to systems that present mixed reality to 
one or more than two workers (or players). They are not limited to systems that present mixed reality, but they can be 
used for any application that measure the position/attitude of the viewpoint of an imaging apparatus by means of 
magnetic sensors, Furthennore, they are not limited to the measurement of the position/attitude of the player's viewpoint 
described in relation to variation 2, but they can be used for applications that measure the position/attitude of any object 
whose position/attitude relative to the viewpoint of the imaging apparatus Is known. 

<Variation 6> 

[0147] Although the above embodiments use magnetic sensors that employ AC magnetic fields, they are not limited 
50 to such magnetic sensors, but they can be applied to gyro-sensors, ultrasonic sensors, or any other sensors that can 
detect the position/attitude or attitude of a measuring object. 

[0146] Note that the present Invention may be applied to either a system constituted by a plurality of devices (e.g., 
a host computer, an interface device, a reader, a printer, and the like), or an apparatus consisting of a single equipment 
(e.g.. a copying machine, a facsimile apparatus, or the like). 
55 [0149] The objects of the present invention are also achieved by supplying a storage medium, which records a 
program code of a software program that can implement the functions of the above-mentioned embodiments to the 
system or apparatus, and reading out and executing the program code stored In the storage medium by a computer 
(or a CPU or MPU) of the system or apparatus. 
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[01 50] In this case, the program code Itself read out from the storage medium Implements the functions of the above- 
mentioned embodiments, and the storage medium which stores the program code constitutes the present invention. 
[0151] As the storage medium for supplying the program code, for example, a floppy disk, hard disk, optical disk, 
magneto-optical disk, CD-ROM, CD-R, magnetic tape, nonvolatile memory card, ROM, and the like may be used. 
5 [01 521 The functions of the above-mentioned embodiments may be Implemented not only by executing the readout 
program code by the computer but also by some or all of actual processing operations executed by an OS (operating 
system) mnning on the computer on the basis of an instruction of the program code. 

[0153] The present invention includes a product, e.g., a printout, obtained by the image processing method of the 
present invention. 

10 [0154] Furthermore, the present Invention also Includes a case where, after the program codes read from the storage 
medium are written in a function expansion card which Is inserted Into the computer or in a memory provided in a 
function expansion unit which is connected to the computer, CPU or the like contained In the function expansion card 
or unit perfonns a part or entire process in accordance with designations of the program codes and realizes functions 
of the above embodiments. 

15 [0155] In a case where the present Invention is applied to the aforesaid storage medium, the storage medium stores 
program codes corresponding to the flowcharts (FIGS. 6 and/or FIGS. 9 and/or FIGS. 1 0 and/or FIGS. 1 1 and/or FIGS. 
15 and/or FIGS. 17) described in the embodiments. 

[0156] As many apparently widely different embodiments of the present invention can be made without departing 
from the spirit and scope thereof, It is to be understood that the Invention is not limited to the specifte embodiments 

20 thereof except as defined In the appended claims. 

[0157] A view transfonnation matrix that represents the position/attitude of an HMD Is generated based on a signal 
that represents the position/attitude of the HMD (S602). On the other hand, landmari<s and their locations are detected 
based on a captured picture (S604) and a calibration matrix AMc is generated using the detected locations of the 
landmarks (S605). The position/attitude of the HMD is calibrated using the view transformation matrix and calibration 

25 matrix AMc generated by the above processes (8606), a pteture of a virtual object is generated based on external 
parameters that represent the position/attitude of the calibrated HMD, and a mixed reality picture Is generated (S607). 
The generated mixed reality picture is displayed In the display section (S609). 



30 Claims 

1 . An information processing apparatus that derives the calibration infomnation needed to measure the position and/ 
or attitude of a measuring object based on the output values of a position and/or attitude sensor, comprising: 

35 input means for entering infomfiation about a match between the position and/or attitude of said measuring 

object and a predetemnined position and/or attitude; 

acquisition means for acquiring the output values from said position and/or attitude sensor according to the 
input by said input means; and 

operation means for deriving said calibration information, based on said predetemnined position and/or attitude 
40 and the output values of said position and/or attitude sensor acquired by said acquisition means. 

2. The infomnation processing apparatus according to claim 1 . wherein: 

said position and/or attitude sensor has been connected directly or indirectly to said measuring object and the 
45 output values of said sensor are infomiatlon that represents the position and/or attitude of said sensor Itself 

in the sensor coordinate system; and 

said calibration Information contains first coordinate transfonnation Infomtatlon for converting the position and/ 
or attitude of said sensor itself In the sensor coordinate system into the position and/or attitude of said meas- 
uring object in the sensor coordinate system and second coordinate transformation infomnation for converting 
so the position and/or attitude In the sensor coordinate system Into the position and/or attitude in a global coor- 

dinate system. 

3. The information processing apparatus according to claim 1. further comprising guiding means for guiding said 
measuring object to said predetermined position and/or attitude. 



55 



4. The information processing apparatus according to claim 2, wherein: 

the measurement of said position and/or attitude is measurement of position and attitude, and said sensor is 
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a position and attitude sensor; and 

said operation means pertonns the process of determining attitude information among said first coordinate 
transformation infomiation and position Infom^ation among said second coordinate transf omiation infomiation. 

5 5. The information processing apparatus according to claim 2. wherein: 

the measu rement of said position and/or attitude is measurement of only attitude, and said sensor Is an attitude 

sensor; and 

said operation means perfonms the process of detemriining pitch-angle and roil-angle Infonnatlon among said 
10 first coordinate transformation infomiation and yaw-angle infomiation among said second coordinate trans- 

formation information. 

6. The infomnation processing apparatus according to claim 2, wherein: 

15 the measurement of said position and/or attitude is measurement of only attitude, and said sensor is an attitude 

sensor; and 

said operation means perfomis the process of detemilning yaw-angle information among said second coor- 
dinate transformation infomiation. 

20 7. The Information processing apparatus according to claim 1 , wherein said measuring means Is a magnetic sensor. 

8. The information processing apparatus according to claim 1 , wherein said measuring object is the viewpoint of the 
user observing a display device that displays virtual space superimposed over the real space transmitted optically 
through a display screen. 

25 

9. The information processing apparatus according to claim 1 , wherein said measuring object is the viewpoint of an 
Imaging means for capturing real space. 

10. The Information processing apparatus according to claim 8, wherein said guiding means comprises: 

30 

geometry information storage means for storing geometry infomnatlon of an object; 
picture generation means for calculating the two-dimensional appearance of said object expected to be ob- 
served on said display screen when said user observes said display screen in said predetermined position/ 
attitude, based on said predetermined position/attitude of said viewpoint and the geometry infomriation of said 
35 object stored in said geometry Information storage means to generate Its picture; and 

picture presentation means for presenting said picture generated by said picture generation means on said 
display screen; and wherein 

said information processing apparatus moves the viewpoint of said user to said predetenrnined position/attitude 
by moving it such that the image of real space observed through said display screen and the image of said 
40 object displayed on said display screen will match geometrically on said display screen. 

11. The information processing apparatus according to claim 9, wherein said guiding means comprises: 

geometry information storage means for storing geometry information of an object; 

45 picture generation means for calculating the two-dimensional appearance of said object expected to be ob- 

served on said display screen when said Imaging means captures real space in said predetermined position/ 
attitude, based on said predetermined position/attitude of said viewpoint and the geometry Infomiation of said 
object stored in said geometry information storage means, and superimposing its picture over the picture of 
real space captured by said Imaging means to generate a superimposed picture; and 

50 picture presentation means for presenting said superimposed picture generated by said picture generation 

means to the operator; and wherein 

said operator moves the viewpoint of said imaging means to said predetemiined position/attitude by moving 
said imaging means In such a way that the picture of said real space and Image of said object superimposed 
over It will match geometrically on said superimposed picture presented by said picture presentation means. 

55 

12. The Infonnatlon processing apparatus according to any of the claims 1 to 8, wherein said guiding means comprises: 

imaging means for capturing real space, having a viewpoint whose relative position/attitude with respect to 
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said measuring object is known; 

geometry infomiation storage means for storing geometry Information; 

picture generation means for calculating the two-dimensional appearance of said object expected to be ob- 
served on said display screen when said imaging means captures real space In said predetermined position/ 

5 attitude, based on the position/attitude of the viewpoint of said Imaging means determined from said prede- 

teimined position/attitude of said measuring object and said relative position/attitude as well as on the geometry 
information of said object stored In said geometry infomiation storage means, and superimposing its picture 
over the picture of real space captured by said imaging means to generate a superimposed picture; and 
picture presentation means for presenting said superimposed picture generated by said picture generation 

10 means to the operator; and wherein 

said operator moves said measuring object to said predetermined position/attitude by moving said imaging 
means In such a way that the picture of said real space and image of said object superimposed over It will 
match geometrically on said superimposed picture presented by said picture presentation means. 

15 13. The information processing apparatus according to claim 1 0 or 11 , wherein said object contains marlcers placed 
In real space and said geometry information contains position Information of said marlcers, 

14. The Information processing apparatus according to claim 10 or 11, wherein said object has an area or volume in 
real space and said geometry Information contains shape infomiation of the object. 

20 

15. The infomiation processing apparatus according to claim 14, wherein said picture generation means draws a wire 
frame image of said object. 

1 6. The information processing apparatus according to claim 1 0 or 1 1 . wherein said object contains a virtual object to 
25 be superimposed over real space. 

17. A mixed reality presentation apparatus which displays virtual space superimposed over a picture of captured real 
space on a display screen or displays virtual space superimposed over the real space transmitted optically through 
a display screen, based on the output values of a position and/or attitude sensor, comprising: 

30 

an information processing apparatus according to claim 10; and 

switching means for switching between presentation mode that presents mixed reality and derivation mode 
that derives calibration information; wherein 

the calibration information needed for presentation of said mixed reality is derived in said derivation mode and 
35 mixed reality Is presented using the calibration information In said presentation mode. 

18. The mixed reality presentation apparatus according to claim 1 7, wherein said apparatus presents a wire frame of 
maricer locations or objects, etc. in addition to, or In place of, all or part of what Is superimposed In said presentation 
mode. 

40 

19. The mixed reality presentation apparatus according to claim 17, wherein said apparatus shares all or part of the 
objects geometry information used for said derivation mode with said presentation mode. 

20. The mixed reality presentation apparatus according to claim 17, wherein display means is shared between said 
45 derivation mode and said presentation mode. 

21 . An information processing method that derives the calibration infomiation needed to measure the position and/or 
attitude of a measuring object based on the output values of a position and/or attitude sensor that detects the 
position and/or attitude of said measuring object, comprising: 

50 

an input process of entering information about a match between the position and/or attitude of said measuring 
object and a predetermined position and/or attitude; 

an acquisition process of acquiring the output values from said position and/or attitude sensor according to 
the input in said input process; and 
55 an operation process of deriving said calibration information, based on said predetermined position and/or 

attitude and the output values of said position and/or attitude sensor acquired In said acquisition process. 

22. A mixed reality presentation method which displays virtual space superimposed over a picture of captured real 
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space on a display screen or displays virtual space superimposed over the real space transmitted optically through 
a display screen, based on the output values of a position and/or attitude sensor, comprising: 

an information processing method according to daim 21 ; and 
5 switching process of switching between presentation mode that presents mixed reality and derivation mode 

that derives calibration information; wherein 

the calibration information needed for presentation of said mixed reality is derived in said derivation mode and 
mixed reality is presented using the calibration information in said presentation mode. 

10 23. A computer-readable storage medium which stores the program code for executing the infonnation processing 
method according to claim 21 . 

24. A computer-readable storage medium which stores the program code for executing the mixed reality presentation 
method according to claim 22. 

15 
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FIG. 11 
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